Information processing apparatus and non-transitory computer readable medium

ABSTRACT

An information processing apparatus that is used in contact with a user includes a detection unit and a control unit. The detection unit detects a motion of the user in a real space performed without contacting an object that is present in the real space. The control unit instructs the object to execute a specific function in the case where plural motions by users for a common object are detected. The specific function is correlated with the plural motions.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based on and claims priority under 35 USC 119 fromJapanese Patent Application No. 2018-102769 filed May 29, 2018.

BACKGROUND (i) Technical Field

The present disclosure relates to an information processing apparatusand a non-transitory computer readable medium.

(ii) Related Art

There is provided a technique of recognizing a device included in animage captured using a camera built in a portable device operated by auser and displaying a virtual user interface corresponding to therecognized device in an augmented reality space to allow operating thedevice through the virtual user interface. (See Japanese UnexaminedPatent Application Publication No. 2013-172432, for example.)

SUMMARY

The device is operated on the basis of an instruction given throughcontact by the user.

Aspects of non-limiting embodiments of the present disclosure relate tooperating an object that is present in the real space without using anoperation through contact by the user.

Aspects of certain non-limiting embodiments of the present disclosureaddress the above advantages and/or other advantages not describedabove. However, aspects of the non-limiting embodiments are not requiredto address the advantages described above, and aspects of thenon-limiting embodiments of the present disclosure may not addressadvantages described above.

According to an aspect of the present disclosure, there is provided aninformation processing apparatus that is used in contact with a user,including: a detection unit that detects a motion of the user in a realspace performed without contacting an object that is present in the realspace; and a control unit that instructs the object to execute aspecific function in a case where a plurality of motions by users for acommon object: are detected, the specific function being correlated withthe plurality of motions.

BRIEF DESCRIPTION OF THE DRAWINGS

Exemplary embodiments of the present disclosure will be described indetail based on the following figures, wherein:

FIG. 1 illustrates an example of a system according to a first exemplaryembodiment;

FIG. 2 illustrates an example of the hardware configuration of an imageforming apparatus;

FIG. 3 illustrates an example of the hardware configuration of asmartphone;

FIG. 4 illustrates an example of the functional configurationimplemented by a processing circuit section through execution of aprogram;

FIG. 5 illustrates an example of a table provided to operate the imageforming apparatus;

FIG. 6 is a flowchart illustrating a portion of an example of processingoperation executed by the processing circuit section;

FIG. 7 is a flowchart illustrating the remaining portion of the exampleof processing operation executed by the processing circuit section;

FIG. 8 illustrates a scene in which a user B as the image capturercaptures an image of his/her own index finger as superposed on the imageforming apparatus on a screen;

FIG. 9 illustrates a scene in which the user captures an image in whichthe user makes operation to move his/her index finger forward in the airwithout contacting the image forming apparatus;

FIG. 10 illustrates a different method of notifying the user ofreception of an operation;

FIG. 11 illustrates a different method of notifying the user ofreception of an operation;

FIG. 12 illustrates a different method of notifying the user ofreception of an operation;

FIG. 13 illustrates a method in which an index finger image is notdisplayed on a liquid crystal display;

FIG. 14 illustrates a different method in which an index finger image isdisplayed on the liquid crystal display;

FIG. 15 illustrates an example of an operation performed throughcooperation between the user B and a user C;

FIG. 16 illustrates a scene in which the user C who cooperates with theuser B performs an operation;

FIG. 17 illustrates a scene in which the user C who cooperates with theuser B performs an operation;

FIG. 18 illustrates variations that appear on the screen of a smartphoneoperated by the user C after authentication by the user B is completed;

FIG. 19 illustrates an example of a system according to a secondexemplary embodiment;

FIG. 20 illustrates an example of a table that is used to operate adoor;

FIG. 21 illustrates a scene in which a user A or the user B as theoperator captures an image of his/her own right hand as superposed on aknob image;

FIGS. 22A and 22B illustrate an example of an operation for a case wherethe operator is the user A alone;

FIGS. 23A and 23B illustrate an example of an operation for a case wherethe operator includes two persons, namely the user A and the user C,with FIG. 23A illustrating a scene before an operation by a gesture andwith FIG. 23B illustrating a scene after an operation by a gesture;

FIGS. 24A and 24B illustrate an example of an operation for a case wherethe operator includes two persons, namely the user A and the user B,with FIG. 24A illustrating a scene before an operation by a gesture andwith FIG. 24B illustrating a scene after an operation by a gesture;

FIGS. 25A and 25B illustrate an example of an operation for a case wherethe operator includes two persons, namely the user A and the user B,with FIG. 25A illustrating a scene before an operation by a gesture andwith FIG. 25B illustrating a scene after an operation by a gesture;

FIG. 26 illustrates an example of a system according to a thirdexemplary embodiment;

FIG. 27 illustrates an example of the hardware configuration of a safe;

FIG. 28 illustrates an example of a table that is used to operate thesafe;

FIG. 29 illustrates an example in which the user B captures an image ofhis/her own right hand as superposed on a right dial lock image;

FIG. 30 illustrates a different example in which the user B captures animage of his/her own right hand as superposed on the right dial lockimage;

FIG. 31 illustrates a different example in which the user B captures animage of his/her own right hand as superposed on the right dial lockimage;

FIG. 32 illustrates a case where the image capture target is an imagedisplayed on a display surface of a display apparatus;

FIG. 33 illustrates an example in which an object captured by a cameraother than the smartphone is operated by a gesture;

FIG. 34 illustrates an example of a system that uses a wearableinformation processing apparatus;

FIG. 35 illustrates a case where an instruction to move a potted plantis provided by a gesture;

FIG. 36 illustrates the hardware configuration of a robot; and

FIG. 37 illustrates an example in which a plurality of users cooperatewith each other to provide the robot with an instruction to move thepotted plant.

DETAILED DESCRIPTION

Exemplary embodiments of the present disclosure will be described belowwith reference to the drawings.

First Exemplary Embodiment

<System Example>

FIG. 1 illustrates an example of a system according to a first exemplaryembodiment.

The system illustrated in FIG. 1 includes an image forming apparatus 100that forms an image on a recording material (hereinafter occasionallyreferred to representatively as “paper”), and two smartphones 200A and200B that capture an image of the image forming apparatus 100.

The image forming apparatus 100 is an example of an object that ispresent in the real space as the operation target.

The image forming apparatus 100 according to the present embodiment hasa copy function of generating a duplicate, a scan function of reading animage of a document, a facsimile transmission/reception function oftransmitting and receiving a facsimile image to and from another device,a print function of recording an image on paper, etc. The copy functionis implemented by combining the scan function and the print function.

It is not necessary that the image forming apparatus 100 should have allof the copy function, the scan function, and the facsimiletransmission/reception function, and the image forming apparatus 100 maybe an apparatus that specializes in one of such functions such as acopier, a scanner, a facsimile transmitter/receiver, or a printingapparatus, for example.

The image forming apparatus 100 may also be an apparatus that has acombination of several functions selected from such functions.

The image forming apparatus 100 includes a user interface section 110that is used to receive an operation from a user and present variouskinds of information to the user, an image reading unit 120 that readsan image of a document, an image forming unit 130 that records an imageon paper, and a housing section 140 that houses the paper.

The smartphones 200A and 200B are each a portable terminal apparatus,and include a camera for image capture, a display screen, acommunication function, etc. as discussed later.

The smartphones 200A and 200B are each an example of an informationprocessing apparatus.

In the case of the present embodiment, the smartphone 200A is operatedby a user A, and the smartphone 200B is operated by a user B.

Images 100A and 100B of the image forming apparatus 100 are displayed onthe respective display screens of the smartphones 200A and 200Billustrated in FIG. 1.

In the following, the smartphones 200A and 200B will occasionally bereferred to collectively as smartphones 200.

<Configuration of Image Forming Apparatus>

FIG. 2 illustrates an example of the hardware configuration of the imageforming apparatus 100.

Portions in FIG. 2 corresponding to those in FIG. 1 are denoted by thecorresponding reference numerals.

The user interface section 110 includes a display section 111 that isused to display an operation screen etc., and an operation receivingsection 112 that receives an operation input by the user.

The display section 111 includes a liquid crystal display panel, anorganic electro-luminescence (EL) display panel, or the like, forexample.

The operation receiving section 112 includes a button, a touch pad, orthe like that receives an operation input by the user.

The image reading unit 120 includes an image reading section that readsan image from a document, and a document transport section thattransports the document to the image reading section. The image readingunit 120 is disposed on top of the image forming unit 130.

The image forming unit 130 includes a micro processing unit (MPU) 131that functions as a control section, a read only memory (ROM) 132, and arandom access memory (RAM) 133.

The MPU 131 provides various kinds of functions by executing a programsuch as firmware read from the ROM 132. The RAM 133 is used as a workarea for the program.

Besides, the image forming unit 130 includes a storage section 134 thatis used to store image data etc., an image processing section 135 thatperforms image processing such as color correction and tone correctionon the image data, an image forming section 136 that forms an image onpaper using an electrophotographic system or an inkjet system, acommunication section 137 that is used for external communication, etc.

The storage section 134 includes a non-volatile storage medium such as ahard disk apparatus or a semiconductor memory. The storage section 134stores image data on a document read by the image reading unit 120 andimage data received through the communication section 137.

The image processing section 135 includes a dedicated processor orprocessing board, for example.

The communication section 137 includes a local area network (LAN)interface or an interface for a facsimile communication network service.The LAN interface is used for communication with the smartphone 200 (seeFIG. 1) discussed earlier.

The MPU 131, the ROM 132, the RAM 133, the storage section 134, theimage processing section 135, the image forming section 136, and thecommunication section 137 are connected to each other through a bus 138.The user interface section 110 and the image reading unit 120 discussedearlier are connected to the bus 138 through an interface 139.

<Configuration of Smartphone>

FIG. 3 illustrates an example of the hardware configuration of thesmartphone 200.

The smartphone 200 includes a processing circuit section 210 thatprocesses data, a non-volatile memory 221 that stores a program anddata, an audio circuit 222 that receives and outputs an audio signal, aliquid crystal display (LCD) 223 that displays an image, a power sourcecontrol device 224, a camera 225 that captures an image, a touch pad 226that detects contact with an operation surface, a Wi-Fi (WirelessFidelity) module 227 that transmits and receives a wireless signal thatconforms to the Wi-Fi (registered trademark) standard, a Bluetooth(registered trademark) module 228 that transmits and receives a wirelesssignal that conforms to the Bluetooth (registered trademark) standardwhich is one of near-field communication standards, and an aerialtactile feel module 229 that gives an artificial tactile feel to aportion of a body that is not in contact with an object.

In the case of the present exemplary embodiment, a surface provided withthe camera 225 and a surface provided with the liquid crystal display223 are a front surface and a back surface, respectively.

This enables the user to capture an image of an object (e.g. the imageforming apparatus) that is present in the real space and a part (e.g. afinger, a hand, or a foot) of his/her own body using the camera 225while confirming the image on the liquid crystal display 223.

In the case where a plurality of cameras 225 are provided or in the casewhere a plurality of liquid crystal displays 223 are provided, it isonly necessary that at least a set of a camera 225 and a liquid crystaldisplay 223 should be disposed on the front surface and the backsurface, respectively.

In the present embodiment, the liquid crystal display 223 is used.However, a different display unit such as an organicelectro-luminescence (EL) display may also be used.

The aerial tactile feel module 229 according to the present embodimentincludes an ultrasonic transducer array in which a plurality ofultrasonic transducers are arranged in a grid shape, for example. Thistype of the aerial tactile feel module 229 is capable of generating thefocal point of ultrasonic waves at a desired position in the air.

A tactile feel perceived by the users varied by adjusting thedistribution of the focal points or the intensity of vibration.

For example, the generated tactile feel may be varied in accordance withthe function as the operation target. A tactile feel is generated in thecase where user motion is detected, in the case where an operation for aspecific function is received, or the like, for example. Motion of adifferent user and the intensity of such motion may be transmittedthrough a tactile feel. The aerial tactile feel module 229 is an exampleof an application unit that applies a tactile feel.

The processing circuit section 210 includes a memory control section 211that controls reading and writing of data from and into the memory 221,a plurality of central processing unit (CPU) cores that executeprocessing, a power source management section 213 that manages supply ofpower, a system management section 214 that manages operation of abuilt-in system, an audio processing section 215 that processes an audiosignal, a graphics processing unit (GPU) 216 that processes an image inreal time, a display control section 217 that displays an image on theliquid crystal display 223, an external interface (I/F) 218 that is usedfor connection with an external module, and a baseband circuit 219 thatprocesses a baseband signal.

FIG. 4 illustrates an example of the functional configurationimplemented by the processing circuit section 210 through execution of aprogram.

The processing circuit section 210 here is an example of the detectionunit, and is also an example of the control unit.

The processing circuit section 210 functions as a gesture detectionsection 251 that detects a gesture by the user by processing image data,an operation target detection section 252 that detects an operationtarget object, a target function acquisition section 253 that acquiresinformation on a function of the object from the operation target object(the image forming apparatus 100 in the present embodiment), anoperation receiving section 254 that receives a gesture by the user asan operation, an instruction transmitting section 255 that transmits aninstruction corresponding to the received operation to the targetobject, a notification section 256 that notifies the user of receptionof the operation, and a cooperation section 257 that cooperates with anoperation by a different user for an identical (common) object.

The gesture detection section 251 according to the present embodimentdetects motion of a part (e.g. a finger, a hand, or a foot) of the bodyof the user as a gesture from an image captured by the camera 225, forexample. The meaning of motion differs for each object detected as theoperation target. Even the meaning of motion for the same object differsamong different functions as the operation target.

Examples of the gesture according to the present embodiment includepushing, pulling, rotating, grasping, kicking, stepping on, clicking,double-clicking, dragging, tapping, flicking, and swiping. The gesturemay be specified using a deep learning function.

A different method may be employed to detect a gesture. For example, agesture may be detected by a Time-of-Flight (ToF) method in which thedistance to an object is calculated by measuring the time untilsemiconductor laser or light emitted from a light emitting diode (LED)is returned after being reflected by the object for each pixel, aStructured Light (SL) chronological pattern projection method in whichthe distance to an object, onto which a vertical stripe pattern that isvaried chronologically is projected, is calculated from variations inbrightness that appear in pixels obtained by capturing images of theobject, a method of calculating the distance to an object usingultrasonic waves or millimeter waves, etc., either singly or incombination thereof. The combination may include a technique ofrecognizing a gesture by processing captured images.

The operation target detection section 252 according to the presentembodiment detects an object as the operation target, an operationelement thereof, etc. by analyzing an image displayed on the liquidcrystal display 223 (see FIG. 3).

Examples of the operation element according to the present embodimentinclude a physical structure (e.g. a button and a switch) to be operatedand display (e.g. a button and a switch) on a screen reproduced in asoftware manner.

The image that is used in the processing is not limited to real-timeimage data acquired from the camera 225 provided to the smartphone 200,and may be mage data acquired from the memory 221 or acquiredexternally.

In the case where a plurality of objects are included in the screen, theoperation target detection section 252 may detect an object, an image ofwhich displayed on the liquid crystal display 223 is closer to thecenter of the screen, as the target, or may detect an object, an imageof which is superposed on an image of a part of the body of the user, asthe operation target, for example.

The operation target detection section 252 may detect a device selectedby the user from a list of devices being connected with the smartphone200 through a wireless LAN or the like as the operation target.

The target function acquisition section 253 according to the presentembodiment acquires information on an operation element that is operablethrough communication with an object detected by the operation targetdetection section 252, for example. For example, the target functionacquisition section 253 acquires information on an operation elementusing a wireless LAN.

The target function acquisition section 253 may also acquire informationon an operation element that is operable from each of communicabledevices, for example.

The operation target is not limited to an object that has acommunication function. Examples of such an operation target include aplant and a natural object. Also in this case, the target functionacquisition section 253 may acquire information on an operation that isexecutable through communication with a different device such as a robotthat is capable of making motion on behalf of a person.

The operation receiving section 254 according to the present embodimentreceives the gesture which is detected by the gesture detection section251 as an operation for a specific function of the object which isdetected by the operation target detection section 252.

The operation receiving section 254 receives an operation with referenceto a table 254A prepared for each object, for example. The table 254Amay be prepared for each function. In the case of FIG. 4, the table 254Ais stored in the operation receiving section 254. The table 254A may bestored in the memory 221 (see FIG. 3) which is connected to theprocessing circuit section 210.

FIG. 5 illustrates an example of a table 254A1 provided to operate theimage forming apparatus 100 (see FIG. 1).

The table 254A1 stores the relationship between operations assumed forthe image forming apparatus 100 and individual users.

The content of the table 254A1 may be registered or corrected by anauthorized user or manager.

The assumed operations are not limited to functions that the imageforming apparatus 100 as the operation target has or functions that areexecutable by the image forming apparatus 100. The operations mayinclude an operation to move the image forming apparatus 100 by applyingan external force and an operation to open and close a movable portionsuch as a lid or a drawer, for example.

In the case of the present embodiment, the table 254A1 stores, for eachuser, for example, whether or not the user is permitted to execute aspecific operation, whether or not cooperation with a different user isrequired, the content of an operation required in the case where suchcooperation is required, etc.

Examples of an operation that requires cooperation with a different userinclude an operation required to be executed at the same timing, anoperation required to be executed temporally successively, and anoperation that is one of a plurality of elements obtained by decomposingone operation. In the case where an operation is required to be executedtemporally successively, it is necessary that an operation by acooperating user should be executed within a time determined in advance.

The different user is not limited to one user, and may be a plurality ofusers.

The order of execution by a plurality of users required to cooperatewith each other may be prescribed, although such an order not prescribedin the table 254A1. In case where the order of execution is prescribed,the operation receiving section 254 (see FIG. 4) does not instructexecution of an operation unless the users execute the operation in theprescribed order.

In the case of FIG. 5, for example, a user A is able to perform anoperation for functions related to printing and facsimile transmissionalone, but is not able to perform an operation for functions related tochanging the color tint for printing (printing color tint) etc.

A user B is able to perform an operation for functions related toprinting, facsimile transmission, changing the printing color tint, andauthentication alone, but requires authentication by a user D to performan operation for functions related to changing the address book and tonecorrection, for example.

A user C requires authentication by the user B to perform an operationfor functions related to printing, facsimile transmission, and changingthe printing color tint, and requires authentication by the user D toperform an operation for functions related to changing the address bookand tone correction, for example. The user C is not able to perform anoperation for a function related to authentication.

The user D is able to perform an operation for functions related toprinting, changing the address book, tone correction, and authenticationalone, but requires input of an area code by the user A to perform anoperation for a function related to facsimile transmission, and requiresauthentication by the user B to perform an operation for a functionrelated to changing the printing color tint, for example.

While four users are given in the example in big the number of users isnot limited to four. The operations are also not limited to the sixtypes given in FIG. 5.

As indicated in FIG. 5, if the combination of users is different, theallocation of operations required from the individual users is alsodifferent.

For example, the allocation of operations to the user B and the user Cand the allocation of operations to the user B and the user D aredifferent from each other.

In the example in FIG. 5, examples of the cooperation for an operationinclude an example in which one user enables the other user to performan operation through authentication, and an example in which the userstake part in inputting a number that specifies the destination offacsimile transmission. However, the users may take part in setting anitem related to a function, for example.

FIG. 4 is described again.

The instruction transmitting section 255 according to the presentembodiment transmits an instruction corresponding to the receivedoperation to the target object (e.g. the image forming apparatus). Inaddition, the instruction transmitting section 255 has a function ofacquiring a response to an instruction (e.g. reception of aninstruction) from the target object.

The notification section 256 according to the present embodiment makesvarious kinds of notifications that assist in performing an operation bya gesture through the audio circuit 222 or the liquid crystal display223.

The cooperation section 257 according to the present embodimentexchanges information on an operation with a smartphone 200 operated bya different user prescribed as a cooperation partner in the table 254A.The cooperation section 257 may also exchange information on an objectas the operation target with a different smartphone 200 that iscommunicable with the cooperation section 257, even if such a smartphone200 is not prescribed as a cooperation partner.

<Processing Operation Performed by Processing Circuit Section>

An example of processing operation executed by the processing circuitsection 210 (see FIG. 4) will be described below with reference to FIGS.6 and 7.

FIG. 6 is a flowchart illustrating a portion of an example of processingoperation executed by the processing circuit section 210. In FIG. 6,steps are denoted by the symbol S. The same also applies to FIG. 7.

The processing circuit section 210 according to the present embodimentrecognizes a part (e.g. a finger) of a human body that is captured inthe screen of the liquid crystal display 223 (see FIG. 3) (step 101).

Next, the processing circuit section 210 detects motion of therecognized part of the human body (step 102).

The above processes are executed by the gesture detection section 251(see FIG. 4), for example. As discussed earlier, a gesture may bedetected not only through an image recognition process, but also usinginformation (a so-called distance image) on a distance measured by asensor.

After that, the processing circuit section 210 specifies a portion of anobject superposed on an image of the part of the human body in thescreen (step 103). This process is executed by the operation targetdetection section 252 (see FIG. 4), for example.

The processing circuit section 210 also recognizes an operation targetobject in advance or concurrently.

In the case where an operation target object may not be specified, theprocessing circuit section 210 makes a notification that an operationtarget object may not be specified, that an image that includes a modelor the like should be captured, etc. This process is executed by thenotification section 256 (see FIG. 4), for example.

Subsequently, the processing circuit section 210 detects the detectedmotion of the human body as an operation on the specified portion (step104). This process is executed by the operation receiving section 254,for example.

In the case where no function is correlated with the specified portionor in the case where the detected motion is not related to execution ofthe correlated function, the processing circuit section 210 does notdetect the detected motion as an operation.

Next, the processing circuit section 210 determines whether or not theuser who is operating the smartphone 200 is authorized for the detectedoperation (step 105).

In the case where a negative result is obtained in step 105, theprocessing circuit section 210 makes a notification that the operationmay not be performed (step 106), and ends the process. The notificationis executed by the notification section 256, for example.

In the case where a positive result is obtained in step 105, theprocessing circuit section 210 determines whether or not execution ofthe detected operation by the user requires cooperation with a differentuser (step 107). This process is executed by the operation receivingsection 254 with reference to the table 254A (see FIG. 4).

In the case where a positive result is obtained in step 107, theprocessing circuit section 210 determines whether or not a condition forcooperation is met (step 108). This process is executed by the operationreceiving section 254 with reference to the table 254A. The processingcircuit section 210 repeatedly makes the determination until thecondition is met (while a negative result is obtained in step 108). Itshould be noted, however, that in the case where the condition is notmet even when a time determined in advance elapses, a notification ofthe content of an operation that requires cooperation may be made, anotification may be sent to the smartphone 200 which is operated by auser, cooperation with whom is required, or a notification that anoperation may not be performed may be made.

In the case where a negative result is obtained in step 107, on theother hand, the processing circuit section 210 determines whether or notthere is any detected operation of the specific portion by a differentuser (step S109). This process is executed by the operation receivingsection 254. This process is not always necessary, and may be used tosynthesize an external force based on instructions from individual usersin the case where a physical force acts on an object, for example.

In the case where a positive result is obtained in step 109, theprocessing circuit section 210 combines operations by a plurality ofusers (step 110). This process is executed by the operation receivingsection 254. Combining of operations may be simple combining, or may beweighted combining.

In the case where a positive result is obtained in step 108, in the casewhere a negative result is obtained in step 109, or after execution ofstep 110, the processing circuit section 210 executes the processillustrated in FIG. 7.

FIG. 7 is a flowchart illustrating the remaining portion of the exampleof processing operation executed by the processing circuit section 210.

The processing circuit section 210 determines whether or not the targetobject is controllable through communication (step 111). This process isalso executed by the operation receiving section 254, for example.

In the case where a positive result is obtained in step 111, theprocessing circuit section 210 transmits a signal that instructsexecution of the operation to the target object (step 112).

In the case where a negative result is obtained in step 111, theprocessing circuit section 210 determines whether or not there is anyself-running apparatus that is capable of making an operation on behalf(step 113). Examples of the self-running apparatus that is capable ofmaking an operation on behalf include a robot.

In the case where a positive result is obtained in step 113, theprocessing circuit section 210 transmits a signal that instructsexecution of the operation to the self-running apparatus (step 114).

In the case where a negative result is obtained in step 113, theprocessing circuit section 210 makes a notification that the operationmay not be performed (step 115).

The processing circuit section 210 ends the process after such steps.

<Operation Examples>

Operation examples implemented using the functions discussed earlierwill be described below.

<Operation Example 1>

FIG. 8 illustrates a scene in which the user B as the image capturercaptures an image of his/her own index finger 301 as superposed on theimage forming apparatus 100 on the screen.

When capturing an image, the user B holds the smartphone 200B withhis/her left hand. The index finger 301 is an index finger of the userB, and is not in contact with the image forming apparatus 100.Specifically, the index finger 301 is positioned in the air between theimage forming apparatus 100 and the smartphone 200B.

In the case of FIG. 8, a portion of the user interface section 110 isdisplayed as enlarged on the liquid crystal display 223 of thesmartphone 200B.

This display state is obtainable by moving two fingers in contact withthe screen of the liquid crystal display 223 away from each other withan image of the image forming apparatus 100 displayed on the liquidcrystal display 223. Display may be enlarged without contacting theliquid crystal display 223 if a voice recognition technique is used.

In the case of FIG. 8, an image 270 (a button 271 for facsimiletransmission and a button 272 for specification setting/registration) inwhich a portion of the user interface section 110 that is present in thereal space is captured and an index finger image 301A are displayed onthe liquid crystal display 223. This screen is an example of an imagefor work.

In the case of the present embodiment, the printing color tint ischangeable when an operation to click on the button 272 forspecification setting/registration is received. In the case of the table254A1 (see FIG. 5), the user B is able to perform an operation to changethe printing color tint alone.

In the real space, as discussed earlier, the image forming apparatus100, the index finger 301 of the user, and the smartphone 200B arepositioned sequentially in this order from the farther side.

Thus, the index finger image 301A captured by the smartphone 200B isdisplayed as superposed on the closer side of the user interface sectionimage 270.

To operate the image forming apparatus 100 using motion of the indexfinger 301, it is necessary that the image forming apparatus 100 shouldbe specified in advance as the operation target.

To specify the image forming apparatus 100, it is possible to use atechnique of specifying a model by processing an image of the imageforming apparatus 100, a technique of detecting character informationsuch as a model number included in an image, a technique of allowingselecting one from candidates displayed on the liquid crystal display223, or the like, for example. The candidates are given as a list ofdevices being connected with the smartphone 200B through a wireless LANor the like, for example.

It is assumed that the image forming apparatus 100 has already beenspecified as the operation target.

In the case of FIG. 8, two operation elements button 271 for facsimiletransmission and the button 272 for specification setting/registration)that are operable are displayed on the liquid crystal display 223.However, three or more operation elements may be displayed, or only oneoperation element may be displayed.

In the case of FIG. 8, two operation elements are displayed on theliquid crystal display 223, and thus it is necessary to move the indexfinger 301 in the air such that the operation element as the operationtarget and the index finger image 301A are superposed on each other.

Movement of the index finger 301 in the air is also detected by thesmartphone 200B as motion of the user. It should be noted, however, thatmotion in the horizontal direction is not correlated with motion of anoperation on the operation element captured in an image.

Therefore, motion made before the index finger image 301A is superposedon the button 272 for specification setting/registration is not receivedas an operation on the operation element.

In the present embodiment, motion of the index finger image 301A isdetected since the operation target includes buttons. However, a portionof the body as the detection target may be determined in accordance withthe operation target. For example, a part of a human body such as ahand, an arm, or a foot may be determined as the detection target.

FIG. 9 illustrates a scene in which the user captures an image in whichthe user performs operation to move his/her index finger 301 forward inthe air without contacting the image forming apparatus 100. Portions inFIG. 9 corresponding to those in FIG. 8 are denoted by the correspondingreference numerals.

The pushing operation here refers to motion in the direction in whichthe index finger 301 is moved away from the smartphone 200B. In otherwords, the pushing operation refers to motion toward the image formingapparatus 100.

Specifically, the pushing operation corresponds to motion to push anoperation element displayed on the user interface section 110 in theactual space. This is because the direction of the line of sight of theuser and the direction in which the camera 225 (see FIG. 3) captures animage coincide with each other.

The index finger image 301A is superposed on the button 272 forspecification setting/registration on the liquid crystal display 223illustrated in FIG. 9, and thus the smartphone 200B detects the pushingoperation as an operation to push the button 272 for specificationsetting/registration.

In the case of the present embodiment, the smartphone 200B stores therelationship between the operation elements and motion in the memory 221(see FIG. 3). The memory 221 may store the relationship between some ofthe operation elements provided to the image forming apparatus 100 andmotion.

The relationship between images of the operation elements provided tothe image forming apparatus 100 and motion recognized as operations maybe stored in the image forming apparatus 100, and a process ofspecifying an operation element as the operation target may be executedby the image forming apparatus 100.

In FIG. 9, the smartphone 200B is able to communicate with the imageforming apparatus 100, and thus transmits a signal that instructsoperation of the button 272 for specification setting/registration tothe image forming apparatus 100.

The smartphone 200B according to the present embodiment is prepared fora function of notifying the user of reception of an operation.

Therefore, in FIG. 9, the user is notified of reception of an operationby varying the manner of display of the button 272 for specificationsetting/registration.

Examples of the method of varying the manner of display includeincreasing the brightness of a region in which the button as theoperation target is displayed, flashing the region, and varying theregion to a different color.

In FIG. 9, vibration is applied to the index finger 301 by ultrasonicwaves concentrated on the surface of the index finger 301. Theultrasonic waves are output from the aerial tactile feel module 229 (seeFIG. 3). The vibration gives the user B a tactile feel that theoperation of the button has been received. When the tactile feel isgiven, the user is informed that his/her own motion has been received asan operation.

The aerial tactile feel module 229 may be provided to the image formingapparatus 100. In this case, the aerial tactile feel module 229 may givea tactile feel to the ball of the index finger 301 which is directedtoward the image forming apparatus 100.

The aerial tactile feel module 229 may be disposed in a space as anindependent apparatus.

<Operation Example 2>

FIG. 10 illustrates a different method of notifying the user ofreception of an operation.

Portions in FIG. 10 corresponding to those in FIG. 9 are denoted by thecorresponding reference numerals.

In the case of FIG. 10, a “click” sound is output from a speaker (notillustrated) although display of the button 272 for specificationsetting/registration is not varied.

A speech such as “Operation has been received” may be generated in thenotification method which uses a sound.

<Operation Example 3>

FIG. 11 illustrates a different method of notifying the user ofreception of an operation.

Portions in FIG. 11 corresponding to those in FIG. 9 are denoted by thecorresponding reference numerals.

In the case of FIG. 11, a small screen 280 that indicates reception ofan operation is additionally displayed although display of the button272 for specification setting/registration is not varied.

A character string “Operation has been received” is written in thescreen 280 illustrated in FIG. 11. As a matter of course, this characterstring is exemplary.

<Operation Example 4>

FIG. 12 illustrates a different method of notifying the user ofreception of an operation.

Portions in FIG. 12 corresponding to those in FIG. 9 are denoted by thecorresponding reference numerals.

In the case of FIG. 12, the manner of display of the index finger image301A has been varied from the manner before reception of an operationalthough display of the button 272 for specificationsetting/registration is not varied.

Examples of the method of varying the manner of display includeincreasing the brightness of a region in which the index finger image301A is displayed, flashing the region, and varying the region to adifferent color.

<Operation Example 5>

FIG. 13 illustrates a method in the index finger image 301A (see FIG. 9)is not displayed on the liquid crystal display 223.

Portions in FIG. 13 corresponding to those in FIG. 9 are denoted by thecorresponding reference numerals.

In the method in which a part of a human body that has entered the rangeof an image captured by the camera 225 (see FIG. 3) is always displayed,for example, a large portion of an operation element may be hidden bythe part of the human body. For example, characters written on theoperation element may be hidden by an image of a hand or an arm.

In such a case, the user may not be able to confirm the operationelement as the operation target on the screen, or may be able to confirmonly a part thereof, which may hinder an operation.

If characters written on an operation element may not be confirmed, theuser may not be able to see what operation element he/she is operating,and may perform an erroneous operation.

Thus, in FIG. 13, the manner of display of an operation element (thebutton 272 for specification setting/registration) at a positionsuperposed by the index finger image 301A is varied to assistconfirmation by the user,

<Operation Example 6>

FIG. 14 illustrates a different method in which the index finger image301A is displayed on the liquid crystal display 223.

Portions in FIG. 14 corresponding to those in FIG. 11 are denoted by thecorresponding reference numerals.

While the index finger image 301A is not displayed at all in OperationExample 5, it is also conceivable that there is a request to confirm aposition on the screen.

Operation Example 6 provides a display method provided in considerationof such a request, and provides the contour line of a region in whichthe index finger image 301A is positioned. The contour line here is anexample of the notation that suggests a portion of the body of the user.

In FIG. 14, the contour line of the index finger image 301A is indicatedby the broken line. However, a transparent image obtained through imageprocessing may be displayed in place of an actual image. The transparentimage here is also an example of the notation that suggests a portion ofthe body of the user.

Only an index finger portion may be selectively displayed by a contourline or as a transparent image while other portions are displayed as anactual image. On the contrary, only an index finger portion may beselectively displayed as an actual image. The area which is hidden by animage of a part of a human body reduced, and thus an image of an objectas the operation target may be confirmed easily.

<Operation Example 7>

While a button that is operable by the user B as the operator alone isoperated in Operation Examples 1 to 6 discussed earlier, an operationthat requires cooperation will be described below.

FIG. 15 illustrates an example of an operation performed throughcooperation between the user B and the user C. Cooperation between anoperation by the user C to change the printing color tint and anoperation by the user B for authentication will be described in relationto this operation example.

FIG. 16 illustrates a scene in which the user C who cooperates with theuser B performs an operation.

As in the operation examples discussed earlier, the index finger image301A which is moved in the air between the image forming apparatus 100and a smartphone 200C operated by the user C is displayed on the liquidcrystal display 223 of the smartphone 200C.

In FIG. 16, the index finger image 301A is positioned on the button 272for specification setting/registration. According to the table 254A1(see FIG. 5), however, authentication by the user B is required for theuser C to change the printing color tint.

Therefore, a small screen 290 that indicates that reception of anoperation requires authentication by an authorized person isadditionally displayed although display of the button 272 forspecification setting/registration is not varied.

A character string “Authentication by authorized person is required” iswritten in the screen 290 illustrated in FIG. 16. As a matter of course,this character string is exemplary. The specific name of the authorizedperson may be indicated.

FIG. 17 illustrates a scene in which the user B who cooperates with theuser C performs an operation.

A button 311 for authentication and a button 312 for job confirmationare displayed on the liquid crystal display 223 of the smartphone 200Bwhich is operated by the user B. As in the operation examples discussedearlier, the index finger image 301A which is moved in the air betweenthe image forming apparatus 100 and the smartphone 200C is displayed onthe liquid crystal display 223 of the smartphone 200C.

In the case of FIG. 17, the index finger image 301A is superposed on thebutton 311 for authentication. The user B is authorized to operate thebutton 311 for authentication alone, and thus an operation on the button311 for authentication is received as a valid operation.

FIG. 18 illustrates variations that appear on the screen of thesmartphone 200C which is operated by the user C after authentication bythe user B is completed.

The screen 290 (see FIG. 16) has disappeared from the screen illustratedin FIG. 18, and a notification that an operation by the user C has beenreceived as being valid is made through variations in the manner ofdisplay of the button 272 for specification setting/registration andapplication of a tactile feel to the index finger 301.

After that, the user C executes the work of changing the printing colortint through a screen that is used for operation and displayed anew.

Second Exemplary Embodiment

In the exemplary embodiment discussed earlier, the operation target isthe image forming apparatus 100 (see FIG. 1). In the present embodiment,however, the operation target is a door that opens and closes about ahinged portion.

FIG. 19 illustrates an example of a system according to a secondexemplary embodiment.

Portions in FIG. 19 corresponding to those in FIG. 1 are denoted by thecorresponding reference numerals.

The system illustrated in FIG. 19 includes a door 400 and twosmartphones 200A and 200B that capture an image of the door 400.

The door 400 is an example of an object that is present in the realspace as the operation target.

The door 400 according to the present embodiment is provided with anelectronic lock (not illustrated), a knob 401, and a drive mechanism 402that drives to open and close the door 400 through communication withthe smartphone 200 tin FIG. 19, the smartphones 200A and 200B).

In FIG. 19, the subject to be imaged is the door 400, and therefore thesmartphones 200 display a door image 400A.

In the case of the present embodiment, the operation receiving section254 (see FIG. 4) of the smartphone 200 receives an operation by agesture of a user with reference to a table 254A2 (see FIG. 20) storedto operate the door 400.

FIG. 20 illustrates an example of the table 254A2 which is used tooperate the door 400 (see FIG. 19).

The table 254A2 stores the relationship between operations assumed forthe door 400 and individual users

The content of the table 254A2 may also be registered or corrected by anauthorized user or manager.

In the case of the present embodiment, the table 254A2 stores, for eachuser, for example, whether or not the user is permitted to execute aspecific operation, whether or not cooperation with a different user isrequired, the content of an operation required in the case where suchcooperation is required, etc.

In the case of FIG. 20, for example, the user A is able to perform anoperation (opening operation) to open the door 400 and an operation(closing operation) to close the door 400, but is not authorized to lockand unlock the door 400.

Gestures by the user A for an opening operation and a closing operationare detected with a sensitivity of unity magnification.

A sensitivity of unity magnification means that the weighting factor is1.0.

The strength of the force is unknown, and thus the sensitivity isprovided in accordance with the direction of an operation by a gesture,the speed of the operation, the distance over which a portion (e.g. theright hand) of the body used in the operation is moved, etc.

The opening operation and the closing operation by the user A may beperformed alone or through cooperation with a different user.

For example, the user B is able to perform an operation (openingoperation) to open the door 400 with the sensitivity multiplied by 1.5.The sensitivity being multiplied by 1.5 means that the weighing factoris 1.5.

The user B is also able to perform an operation (closing operation) toclose the door 400 with the sensitivity multiplied by two. The user isauthorized to lock and unlock the door 400.

For example, the user C is able to perform an operation (openingoperation) to open the door 400 with the sensitivity reduced to half.

On the other hand, the user C is not permitted to perform an operation(closing operation) to close the door 400. Meanwhile, the user C is ableto lock and unlock the door 400 together with the user B. That the userC is not able to lock and unlock the door 400 alone.

While three users are given in the example in FIG. 20, the number ofusers is not limited to three. The operations are also not limited tothe four types given in FIG. 20.

<Operation Example 1>

FIG. 21 illustrates a scene in which the user A or the user B as theoperator captures an mage of his/her own right hand 310 as superposed ona knob image 401A.

In FIG. 21, the users are not differentiated from each other, and thesmartphones are indicated as the smartphone 200.

Whether the image capturer is the user A or the user B, the imagecapturer captures an image of the door 400 while holding the smartphone200 with his/her left hand as illustrated in FIG. 19. Therefore, thedoor image 400A and the knob image 401A are displayed on the liquidcrystal display 223.

In the case of the present embodiment, the right hand 310 is positionedbetween the knob 401 of the door 400 and the smartphone 200. The righthand 310 is not in contact with the knob 401.

Therefore, a right hand image 310A is displayed on the liquid crystaldisplay 223 as superposed on the door image 400A. In the case of FIG.21, the right hand image 310A is an actual image.

In the case of FIG. 21, the right hand 310 is making motion as holdingand turning the knob 401 in the air between the door 400 and thesmartphone 200.

The door 400 illustrated in FIG. 21 opens toward the farther side whenpushed, and closes when pulled toward the closer side.

Although the knob 401 is attached to the door 400 illustrated in FIG.21, a lever-type handle (lever handle) may be attached to the door 400.

In the case of FIG. 21, a swinging door that opens and closes with adoor surface drawing an arc is assumed. However, the door 400 may alsobe a sliding door guided by a groove or a rail to open and closelinearly, a glide sliding door obtained by combining a swinging door anda sliding door, a folding door, a revolving door, etc.

The door 400 is not limited to a single-swing door, and may be adouble-swing door.

<Operation Example 2>

FIGS. 22A and 22B illustrate an example of an operation for a case wherethe operator is the user A alone, with FIG. 22A illustrating a scenebefore an operation by a gesture and with FIG. 22B illustrating a sceneafter an operation by a gesture.

In the case of FIGS. 22A and 22 b, the user A captures with thesmartphone 200A an image of a gesture of pushing the door 400 open.

In the case of FIG. 20, an operation (opening operation) by the user Ato open the door 400 is permitted. Therefore, an instruction to open thedoor 400 in the farther direction from the user A is transmitted fromthe smartphone 200A to the drive mechanism 402 (see FIG. 21). In FIG.22B, the door 400 is actually open.

<Operation Example 3>

FIGS. 23A and 23B illustrate an example of an operation for a case wherethe operator includes two persons, namely the user A and the user C,with FIG. 23A illustrating a scene before an operation by a gesture andwith FIG. 23B illustrating a scene after an operation by a gesture.

In the case of FIGS. 23A and 23B, both the user A and the user C performan operation (opening operation) to open the door 400. Both the user Aand the user C are permitted to perform an operation (opening operation)to open the door 400. It should be noted, however, that the sensitivityof an operation by the user C is corrected to half.

In this case, the operation receiving section 254 (see FIG. 4) detectsan operation on the knob 401 by a different user (see step 109 in FIG.6), and provides the drive mechanism 402 (see FIG. 21) with aninstruction synthesized from the two operations.

In the example in FIGS. 23A and 23B, the directions of the operations bythe two persons are the same as each other, and thus the operations actin the direction of increasing the speed at which the door 400 isopened.

A tactile feel that matches the speed of an operation by the other useror the magnitude of an acting force may be generated to be given.

<Operation Example 4>

FIGS. 24A and 24B illustrate an example of an operation for a case wherethe operator includes two persons, namely the user A and the user B,with FIG. 24A illustrating a scene before an operation by a gesture andwith FIG. 24B illustrating a scene after an operation by a gesture.

In the case of FIGS. 24A and 24B, the user A is performing an operation(opening operation) to open the door 400, while the user B is performingan operation (closing operation) to close the door 400.

The user A is permitted to perform an operation (opening operation) toopen the door 400, and the user B is permitted to perform an operation(closing operation) to close the door 400. It should be noted, however,that the sensitivity of an operation by the user B is corrected todouble.

Also in this case, the operation receiving section 254 (see FIG. 4)detects an operation on the knob 401 by a different user (see step 109in FIG. 6), and provides the drive mechanism 402 (see FIG. 21) with aninstruction synthesized from the two operations.

In the example in FIGS. 24A and 24B, the directions of the operations bythe two persons are opposite to each other, and the operations afterbeing weighted are balanced with each other. Therefore, the door 400 ismaintained in a stationary state.

Also in this case, a tactile feel that matches the speed of an operationby the other user or the magnitude of an acting force may be generatedto be given.

<Operation Example 5>

FIGS. 25A and 25B illustrate an example of an operation for a case wherethe operator includes two persons, namely the user A and the user B,with FIG. 25A illustrating a scene before an operation by a gesture andwith FIG. 25B illustrating a scene after an operation by a gesture.

Also in the case of FIGS. 25A and 25B, the user A is performing anoperation (opening operation) to open the door 400, while the user B isperforming an operation (closing operation) to close the door 400.

It should be noted, however, that in the case of FIGS. 25A and 25B, theoperation by the user A after being weighted is greater than theoperation by the user B after being weighted. That is, a priority isgiven to the operation (opening operation) to open the door 400.

Therefore, the door 400 is driven to be opened. It should be noted,however, that the speed at which the door 400 is opened is lower thanthat for a case where the user A performs an operation alone.

Third Exemplary Embodiment

In the present embodiment, the operation target is a safe that has twooperation elements.

FIG. 26 illustrates an example of a system according to a thirdexemplary embodiment.

Portions in FIG. 26 corresponding to those in FIG. 1 are denoted by thecorresponding reference numerals.

The system illustrated in FIG. 26 includes a safe 500 and twosmartphones 200A and 200B that capture an image of the safe 500.

The safe 500 is an example of an object that is present in the realspace as the operation target.

A door 501 of the safe 500 illustrated in FIG. 26 is provided with twodial locks 502 and 503 that are not unlocked unless set to a registerednumber.

FIG. 27 illustrates an example of the hardware configuration of the safe500.

The safe 500 includes an MPU 511, a ROM 512 that stores data such asfirmware, a RAM 513 that is used as a work area for a program, acommunication section 514 that is used for communication with anexternal device such as the smartphone 200 (see FIG. 26), and a lockingmechanism 515 that electrically unlocks and locks the safe 500. In thecase of FIG. 27, the various sections are connected to each otherthrough a bus 516. As a matter of course, communication among thevarious sections may be achieved by a component other than the bus 516.

FIG. 28 illustrates an example of a table 254A3 for operation of thesafe 500 (see FIG. 26).

The content of the table 254A3 may be registered or corrected by anauthorized user or manager.

In the case of the present embodiment, the table 254A3 stores, for eachuser, for example, whether or not the user is permitted to execute aspecific operation, whether or not the user is permitted to execute anoperation through cooperation with a different user, etc.

In the case of FIG. 28, for example, an unlocking operation 1 based oncooperation with the user B and an unlocking operation 3 based oncooperation with the user D are allocated to the user A.

The role of the user A in the unlocking operation 1 which is executedthrough cooperation with the user B is to operate the left dial lock502. Therefore, even if the user A operates the right dial lock 503 inthe unlocking operation 1, such an operation is invalid.

The role of the user A in the unlocking operation 3 which is executedthrough cooperation with the user D is to operate the right dial lock503. Therefore, even if the user A operates the left dial lock 502 inthe unlocking operation 3, such an operation is invalid.

Meanwhile, even if the user A operates one of the left dial lock 502 andthe right dial lock 503 in cooperation with a different user (e.g. theuser C), such an operation is invalid.

Only an unlocking operation 2 based on cooperation with the user C isallocated to the user B, for example.

The role of the user B in the unlocking operation 2 which is executedthrough cooperation with the user C is to operate the left dial lock502. Therefore, even if the user B operates the right dial lock 503 inthe unlocking operation 2, such an operation is invalid.

Meanwhile, even if the user B operates one of the dial locks 502 and 503in cooperation with a different user (e.g. the user A or the user D),such an operation invalid.

Only the unlocking operation 2 based on cooperation with the user B isallocated to the user C, for example.

The role of the user C in the unlocking operation 2 which is executedthrough cooperation with the uses B is to operate the right dial lock503. As a matter of course, even if the user C operates the left diallock 502 in the case where the user C cooperates with the user B, suchan operation is invalid. Meanwhile, even if the user C operates one ofthe left dial lock 502 and the right dial lock 503 in cooperation with adifferent user, such an operation is invalid.

Only the unlocking operation 3 based on cooperation with the user A isallocated to the user D, for example.

The role of the user D in the unlocking operation 3 which is executedthrough cooperation with the user A is to operate the left dial lock502. As a matter of course, even if the user D operates the right diallock 503 in the case where the user D cooperates with the user A, suchan operation is invalid. Meanwhile, even if the user D operates one ofthe left dial lock 502 and the right dial lock 503 in cooperation with adifferent user, such an operation is invalid.

Although there is no user that is able to operate both the left andright dial locks 502 and 503 alone in FIG. 28, there may be such a user.

FIG. 28 does not indicate a condition for the timing to operate the leftdial lock 502 and the right dial lock 503. However, an additionalcondition may be provided, such as a condition that the dial locks 502and 503 should be operated at the same time (with a lag of a time lengthdetermined in advance permitted), and a condition as to which of theleft dial lock 502 and the right dial lock 503 should be operated first.The order of operations may be different among combinations of users.

Examples of the operation element that is used for unlocking include acombination of an operation element that requires sequential input ofalphanumeric characters and the dial locks, a combination of a key andthe dial locks, a biometric authentication device (e.g. a fingerprintauthentication device, a vein authentication device, and an irisauthentication device), and a method that uses swipe or patternauthentication.

The operation element that requires sequential input of alphanumericcharacters may be a type that requires input of characters, onecharacter at a time, or a type that requires operation of buttons onwhich alphanumeric characters are printed.

<Operation Example 1>

FIG. 29 illustrates an example in which the user B captures an image ofhis/her own right hand 310 as superposed on a right dial lock image503A.

The user B reaches his/her right hand 310 in the air while holding thesmartphone 200B with his/her left hand (not illustrated), and performs agesture of setting the right dial lock 503 to a number withoutcontacting the right dial lock 503.

The manner of an operation by the user B illustrated in FIG. 29corresponds to the unlocking operation 1 (see FIG. 28) which isperformed in cooperation with the user A. Since this operation is valid,the user B has a tactile feel in his/her right hand 310.

In the case of FIG. 29, the right hand image 310A as an actual image isdisplayed on the liquid crystal display 223 together with a left diallock image 502A, the right dial lock image 503A, and a door image 501A.

The user B performs operation to set the right dial lock image 503A to anumber while seeing the right dial lock image 503A being displayed. Inthe case of FIG. 29, only the right hand image 310A of the user B isdisplayed.

In this event, the user A may be given a tactile feel when the user Bperforms an operation, and the user B may be given a tactile feel whenthe user A performs an operation.

<Operation Example 2>

In the case of Operation Example 1 discussed earlier, the manner of anoperation by a different user may not be confirmed on the screen.Therefore, it is necessary for the user to confirm an operation by adifferent user with the line of sight of the user away from the screen,for example, in the case where it is necessary to match the timing of anoperation by the user himself/herself and the timing of an operation bythe different user, in the case where it is necessary to follow theorder of execution determined in advance, etc.

FIG. 30 illustrates a different example in which the user B captures animage of his/her own right hand 310 as superposed on the right dial lockimage 503A.

Portions in FIG. 30 corresponding to those in FIG. 29 are denoted by thecorresponding reference numerals.

In the case of FIG. 30, the liquid crystal display 223 indicates thatthe cooperating user A is performing an operation through display of avirtual hand 510 correlated with the user A and an arrow 511 thatindicates the direction of an operation on the left dial lock image502A.

In FIG. 30, the fact that the user A is performing an operation isindicated through display of the virtual hand 510. However, such a factmay be indicated using characters, or by varying the manner of displayof the left dial lock image 502A as the operation target. For example,characters Operating may be displayed, the color of the left dial lockimage 502A may be varied, or the left dial lock image 502A may beflashed.

<Operation Example 3>

FIG. 31 illustrates a different example in which the user B captures animage of his/her own right hand 310 as superposed on the right dial lockimage 503A.

Portions in FIG. 31 corresponding to those in FIG. 29 are denoted by thecorresponding reference numerals.

In the case of FIG. 31, a small screen 550 that displays an image beingcaptured by the smartphone 200A which is operated by the cooperatinguser A is displayed on the liquid crystal display 223 in a part of animage being captured by the smartphone 200B which is operated by theuser B.

In the case of the present embodiment, the image which is displayed onthe small screen 550 is received from the cooperating smartphone 200A.

In FIG. 31, a title that indicates that the screen is used by the user Ato perform an operation is displayed on the small screen 550. The smallscreen 550 is an example of an image that displays motion of a differentuser.

In the case of FIG. 31, a safe image 500A, the door image 501A, the leftdial lock image 502A, the right dial lock image 503A, and the right handimage 310A are displayed on the liquid crystal display 223.

Similar images are displayed inside the small screen 550.

In the case of FIG. 31, the directions of operations of the left diallock 502 and the right dial lock 503 are indicated by arrows.

As a matter of course, a similar image is displayed on the liquidcrystal display 223 of the smartphone 200A which is held by thecooperating user A.

Fourth Exemplary Embodiment

In the case of the exemplary embodiment discussed earlier, thesmartphone 200 directly captures an image of an object that is presentin the real space. However, the image capture target may not actuallyexist in the space to be captured in an image.

FIG. 32 illustrates a case where the image capture target is an imagedisplayed on a display surface 601 of a display apparatus 600.

The display apparatus 600 is a television receiver or a monitor, forexample. The display surface 601 of the display apparatus 600 displays auser interface section image 110A of the image forming apparatus 100(see FIG. 1).

The user interface section image 110A here is an image reproduced fromimage data obtained by capturing an image of the image forming apparatus110 which actually exists.

Meanwhile, an image 270 (a button 271 for facsimile transmission and abutton 272 for specification setting/registration) in which a portion ofthe user interface section 110 is captured is displayed on the liquidcrystal display 223 of the smartphone 200. In the case of FIG. 32, theindex finger image 301A is displayed as superposed on the button 272 forspecification setting/registration.

The present embodiment is the same as Operation Example 1 described withreference to FIG. 8 except that the target of image capture by thesmartphone 200 differs from the image forming apparatus 100 (see FIG. 1)as the operation target.

Also in this case, it is possible to operate the image forming apparatus100 in the same manner as in Operation Example 1 if the image formingapparatus 100 as the operation target is specified in advance from alist of devices that are communicable with the smartphone 200.

In the case where the corresponding image forming apparatus 100 may bespecified using an image recognition technique even if the relationshipbetween the user interface section image 110A being displayed on thedisplay surface 601 and the image forming apparatus 100 as the operationtarget is unknown, the specified image forming apparatus 100 may beoperated by a gesture from a remote location.

Fifth Exemplary Embodiment

In the case of the exemplary embodiment discussed earlier, the operationtarget object (or an image thereof) and a part of the body of the userare captured in an image at the same time using the camera 225 (see FIG.3) provided to the smartphone 200. However, an image captured with adifferent camera may be used.

FIG. 33 illustrates an example in which an object captured by a cameraother than the smartphone is operated by a gesture.

Portions in FIG. 33 corresponding to those in FIG. 9 are denoted by thecorresponding reference numerals.

In FIG. 33, a web camera 700 is used as a camera that captures an imageof the image forming apparatus 100.

A smartphone or a different image capture apparatus used by a differentuser may be used in place of the web camera 700.

In the case of the present embodiment, an image of the index finger 301which is used to perform an operation is captured by the camera 225 (seeFIG. 3) of the smartphone 200 which is held by the user as the operator,and displayed as superposed on the image forming apparatus image 100A(specifically, the button 272 for specification setting/registrationwhich is displayed on the user interface section 110).

In the case of the exemplary embodiment, the index finger image 301A ofthe user is extracted from the image which is captured by the smartphone200 to displayed, or a virtual fingertip icon or the like that has beenprepared independently of the captured image is displayed.

In the case of the present embodiment, the smartphone 200 which hasreceived an operation performed along with a gesture by the usertransmits an instruction to the image forming apparatus 100 through anetwork (not illustrated).

Sixth Exemplary Embodiment

In the exemplary embodiment discussed earlier, the smartphone 200 (seeFIG. 1) is described as an example of an information processingapparatus that receives an operation on the age forming apparatus 100 bycapturing an image of a gesture by the user as the operator. However, awearable information processing apparatus may also be used.

FIG. 34 illustrates an example of a system that uses a wearableinformation processing apparatus.

Portions in FIG. 34 corresponding to those in FIG. 1 are denoted by thecorresponding reference numerals.

In the case of FIG. 34, the user wears a glass-type terminal 800. Thehardware configuration (see FIG. 3) and the functional configuration(see FIG. 4) of the glass-type terminal 800 are the same as those of thesmartphone 200. It should be noted, however, that the touch pad 226 (seeFIG. 3) is not attached to the glass-type terminal 800.

In the case of the glass-type terminal 800, the user is able to usehis/her two hands freely. Thus, the glass-type terminal 800 may be usedto input an operation using two hands. For example, the glass-typeterminal 800 may be used to provide an instruction for an operation to ahumanoid robot. Therefore, it is possible to extend the range ofoperations that may be input compared to a case where the smartphone 200is used.

The user in FIG. 34 sees the user interface section image 110A.

Seventh Exemplary Embodiment

While the operation target object which is displayed on the smartphone200 is provided with a communication function in the exemplaryembodiment discussed earlier, a case where the operation target is notprovided with a communication function will be described.

FIG. 35 illustrates a case where an instruction to move a potted plant900 is provided by a gesture.

Potted plant images 900A and 900B are displayed on the respectivescreens of the smartphones 200A and 200B. Also in this case, the righthand 310 in the actual space is moved without contacting the pottedplant 900.

The potted plant 900 is not provided with a communication function or aself-running apparatus. Therefore, unlike the exemplary embodimentdiscussed earlier, even if motion of the right hand is captured in animage by the smartphone 200, the potted plant 900 may not be moved inaccordance with such motion.

In the present embodiment, a humanoid robot 1000 that has a self-runningfunction is designated as the destination of transmission of anoperation by a gesture.

The robot 1000 as the destination of transmission of an instruction maybe designated from a list of devices that are communicable with thesmartphone 200, for example.

Although the robot 1000 illustrated in FIG. 35 has a humanoidappearance, the robot may have any appearance as long as a necessaryfunction is provided. For example, the robot may look like an animalsuch as a dog or a cat, a plant such as a flower or a tree, or a vehicleor a craft such as a car (including a train) or an airplane.

The robot 1000 illustrated in FIG. 35 includes a body portion 1011, ahead portion 1012, arms 1013 and 1015, hands 1014 and 1016, and legs1017 and 1018.

The body portion 1011 stores an electronic component for signalprocessing. The body portion 1011 may be provided with a display deviceor an acoustic device.

The head portion 1012 is coupled to the body portion 1011 via a jointmechanism provided at a neck portion. In the case of the presentembodiment, the joint mechanism is rotatable about three axes. Rotationsabout three axes include yaw (rotation about the z-axis), roll (rotationabout the x-axis), and pitch (rotation about the y-axis), The jointmechanism here is an example of the movable section.

It is not necessary that the joint mechanism should be rotatable aboutall the three axes, and the joint mechanism may be rotatable about onlyone axis or two axes. Such rotation may be implemented by a motor (notillustrated), or may be implemented manually. A case where the headportion 1012 is fixed with respect to the body portion 1011 is notexcluded.

The head portion 1012 is provided with eyes 1012A and 1012B. The eyes1012A and 1012B may be disposed decoratively, or may include an imagecapture device, a projector, a lighting fixture, etc. built therein. Thehead portion 1012 may be provided with movable ears.

The arms 1013 and 1015 according to the present embodiment are coupledto the body portion 1011 via a joint mechanism. An overarm and a forearmof the arms 1013 and 1015 are coupled to each other via a jointmechanism. The joint mechanism here may be of a multi-axis type or asingle-axis type as with that for the head portion 1012. Rotation aboutan axis may be implemented by a motor (not illustrated), or may beimplemented manually. The arms 1013 and 1015 may be fixed to the bodyportion 1011. The joint mechanism here is also an example of the movablesection.

The arms 1013 and 1015 may be bent to an angle determined in advance tobe used to transport an object.

The hands 1014 and 1016 are coupled to the arms 1013 and 1015,respectively, via a joint mechanism provided at a wrist portion. A palmand fingers of the hands 1014 and 1016 are coupled to each other via ajoint mechanism. The joint mechanism here may be of a multi-axis type ora single-axis type as with that for the head portion 1012. Rotationabout an axis may be implemented by a motor (not illustrated), or may beimplemented manually. In the case of the present embodiment, the hands1014 and 1016 may grasp an object by opening and closing the fingers.The joint mechanism here is also an example of the movable section.

The hands 1014 and 1016 may be fixed with respect to the arms 1013 and1015, respectively.

The legs 1017 and 1018 may be coupled to the body portion 1011 via ajoint mechanism, or may be attached to the body portion 1011 as aself-running apparatus such as a wheel or a caterpillar.

In the case where the legs 1017 and 1018 are coupled to the body portion1011 via a joint mechanism, the joint mechanism may be of a multi-axistype or a single-axis type as with that for the head portion 1012.

Rotation about an axis may be implemented by a motor (not illustrated),or may be implemented manually. The legs 1017 and 1018 may be fixed withrespect to the body portion 1011. The joint mechanism here is also anexample of the movable section.

FIG. 36 illustrates the hardware configuration of the robot 1000.

The robot 1000 includes a control section 1021 that controls motion ofthe entire apparatus, a camera 1022 that captures an image of a scenearound the robot, a speaker 1023 that reproduces voices forconversation, tunes, and sound effects, a microphone 1024 that is usedto input or acquire a sound, a movable mechanism 1025 such as a jointmechanism, a communication section 1026 that is used to communicate withan external apparatus (e.g. a smartphone), a display section 1027 thatdisplays an image, a moving mechanism 1028 that moves the entireapparatus, a power source 1029 that supplies electric power to thevarious sections, a sensor 1030 that is used to collect a state of thevarious sections and information on the surroundings, and a positiondetection section 1031 that is used to acquire positional information.Such sections are connected to each other via a bus 1032, for example.

It should be understood that the hardware configuration illustrated inFIG. 36 is exemplary. Thus, it is not necessary that the robot 1000should be provided with all the functional sections discussed earlier.

The robot 1000 may be further provided with a functional section (notillustrated). For example, the robot 1000 may be provided with a powerbutton, a storage apparatus (such as a hard disk apparatus or asemiconductor memory), a heat source (including a cooling source), orthe like.

The control section 1021 is a so-called computer, and includes a CPU, aROM, and a RAM. The ROM stores a program to be executed by the CPU. TheCPU reads the program stored in the ROM, and executes the program usingthe RAM as a work area. The CPU controls operation of the varioussections constituting the robot 1000 through execution of the program.

This program includes a program associated with implementation of analgorithm corresponding to artificial intelligence. The CPU and the RAMconstituting the control section 1021 provide a computational resourcethat is used by the artificial intelligence.

The control section 1021 according to the present embodiment processesinformation acquired through the camera 1022, the microphone 1024, andthe sensor 1030, for example, with the artificial intelligence toautonomously determine operation that matches the surroundingenvironment and the state of the robot 1000.

For example, a voice may be output through the speaker 1023, a messagemay be transmitted through the communication section 1026, and an imagemay be output through the display section 1027.

The control section 1021 may establish communication with the userthrough input and output of such information and motion of the movablemechanism 1025. Examples of application of communication includecustomer attendance .and leading meetings.

The control section 1021 also has a function of collecting additionalinformation through Internet search or communication with an externalcomputer in the case where an unknown situation occurs, and finding asolution in accordance with the degree of similarity to an event foundthe search.

In the case of the present embodiment, examples of the informationacquired by the control section 1021 include information obtainedthrough the senses of sight, hearing, touch, taste, smell, and balanceand the temperature.

The sense of sight is implemented through a process of recognizing animage captured by the camera 1022.

The sense of hearing is implemented through a process of recognizing asound acquired by the microphone 1024.

Examples of the sense of touch include superficial sensation (such astouch, pain, and temperature), deep sensation (such as pressure,position, and vibration), and cortical sensation (such as two-pointdiscrimination and three-dimensional discrimination).

The control section 1021 is able to discriminate differences in thesense of touch.

The senses of touch, taste, smell, and balance and the temperature areimplemented through detection of information by the sensor 1030 ofvarious types. The temperature includes an ambient temperature, aninternal temperature, and the body temperature of a human or an animal.

The information acquired by the control section 1021 may include brainwaves of a human or an animal. In this case, the brain waves may beobtained by the communication section 1026 receiving information sentfrom a brain wave detection device mounted to a human or the like.

In the case of the present embodiment, the camera 1022 is disposed atthe positions of the eyes 1012A and 1012B (see FIG. 35).

In the case where a projector is used as the display section 1027, theprojector may be disposed at one or both of the eyes 1012A and 1012B(see FIG. 1), for example. The projector may be disposed in the bodyportion 1011 or the head portion 1012.

The movable mechanism 1025 may be used not only to transport an objectbut also to express a feeling.

In the case where the movable mechanism 1025 is used to transport anobject, the movable mechanism 1025 implements operation such asgrasping, holding, and supporting the object, for example, throughdeformation of the arms 1013 and 1015 and the hands 1014 and 1016 (seeFIG. 35).

In the case where the movable mechanism 1025 is used to express afeeling, the movable mechanism 1025 implements operation such as tiltingthe head, looking up, looking around, giving a cheer, and pointing afinger through drive of the head portion 1012, the arms 1013 and 1015,the hands 1014 and 1016, etc. (see FIG. 35), for example.

The communication section 1026 according to the present embodimentcommunicates with an external apparatus wirelessly.

The robot 1000 is provided with a number of communication sections 1026,the number corresponding to the number of communication methods used byan external apparatus assumed as the target of communication. Examplesof the communication methods include infrared communication, visiblelight communication, proximity wireless communication, Wi-Fi (registeredtrademark), Bluetooth (registered trademark), RFID (registeredtrademark), ZigBee (registered trademark), IEEE 802.11a (registeredtrademark), MulteFire, and Low Power Wide Area (LPWA).

Bands used for ireless communication include a short-wavelength region(e.g. 800 MHz to 920 MHz) and a long-wavelength region (e.g. 2.4 GHz and5 GHz).

The communication section 1026 and the external apparatus may beconnected to each other using a communication cable.

The display section 1027 may be used to achieve visual communicationwith the user. For example, the display section 1027 may displaycharacters and figures.

In the case where the display section 1027 is disposed at the headportion 1012, the display section 1027 may display a facial expression.

In the case of the present embodiment, a wheel or a caterpillar is usedfor the moving mechanism 1028. However, the robot 1000 may be moved by apneumatic force using a propeller or a compressed air blowout mechanism.

A secondary battery is used for the power source 1029 according to thepresent embodiment. However, any of a primary battery, a fuel cell, anda solar cell that generate electric power may also be used.

Alternatively, the robot 1000 may receive supply of electric power froman external apparatus through a power cable, rather than from the powersource 1029.

In the case of the present embodiment, the robot 1000 is provided withthe position detection section 1031.

The position detection section 1031 utilizes a method in which locationinformation is read from a global positioning system (GPS) signal, anIndoor MEssaging System (IMES) method in which an indoor position ismeasured using a signal that is equivalent to the GPS signal, a Wi-Fiposition measurement method in which a position is measured from theintensity, arrival time, etc. of radio waves transmitted from aplurality of Wi-Fi access points, a base-station position measurementmethod in which a position is measured from the direction and delay timeof a response to a signal regularly generated from a base station, asound-wave position measurement method in which a position is measuredby receiving ultrasonic waves in an inaudible range, a Bluetoothposition measurement method in which a position is measured by receivingradio waves from a beacon that uses Bluetooth, a visible-light positionmeasurement method in which a position is measured using positionalinformation transferred by flashing on and off of illumination lightfrom a light emitting diode (LED) or the like, or an autonomousnavigation method in which the current position is measured using anacceleration sensor, a gyro sensor, etc., for example.

FIG. 37 illustrates an example in which a plurality of users cooperatewith each other to provide the robot 1000 with an instruction to movethe potted plant 900.

Portions in FIG. 37 corresponding to those in FIG. 35 are denoted by thecorresponding reference numerals.

As in the other exemplary embodiments, the user A and the user B movetheir right hand 310 in front of their own smartphones 200A and 200B,respectively, and capture an image of such motion.

When the motion of the right hand 310 is received as an operation, thesmartphone 200 instructs the robot 1000, which is able to move to thepotted plant 900, to execute the operation. The operation here is tomove the potted plant 900 sideways.

This operation for movement does not always require cooperation betweenthe user A and the user B. Therefore, in the case where an operation onthe potted plant 900 by a different user is detected (see step 109 inFIG. 6), the smartphones 200A and 200B provide the robot 1000 with aninstruction obtained by combining the two operations. Operations to becombined may be given different weights different users as in the caseof the second exemplary embodiment discussed earlier.

When the instruction for movement is received, the robot 1000 moves tothe location of the potted plant 900 which is the target of movement,and grasps and moves the potted plant 900 sideways.

In this event, the smartphones 200A and 200B display a robot image 1000Agrasping and moving the potted plant image 900A and the right hand image310A.

In the case of FIG. 37, the right hand image 310A is displayed as atransparent image in order to facilitate confirmation of the pottedplant image 900A and the robot image 1000A.

In the case of FIG. 37, in addition, a small screen 1100 that indicateshow the different user B operates the same object is displayed on thesmartphone 200A which is held by the user A, and a small screen 1200that indicates how the different user A operates the same object isdisplayed on the smartphone 200B which is held by the user B. The smallscreens 1100 and 1200 are an example of an image that displays motion ofa different user.

The small screens 1100 and 1200 also display an arrow that indicates thedirection and the magnitude of an operation by each user.

In the case of FIG. 37, both the potted plant 900 and the robot 1000actually exist in front of the eyes of the users, and thus there islittle need to display the right hand image 310A in a transparentmanner. In the case where the potted plant images 900A and 900B beingdisplayed on the smartphones 200A and 200B, respectively, are notpresent in front of the eyes of the users (e.g. in the case where animage from a web camera is displayed on the smartphone 200), however,the right hand image 310A may be displayed in a transparent manner inorder to facilitate confirmation of a scene in the actual space.

Other Exemplary Embodiments

While exemplary embodiments of the present disclosure have beendescribed above, the technical scope of the present disclosure is notlimited to the exemplary embodiments discussed above. It is apparentfrom the following claims that a variety of modifications andimprovements that may be made to the exemplary embodiments discussedabove also fall within the technical scope of the present disclosure.

For example, in the exemplary embodiments discussed earlier, thesmartphone 200 (see FIG. 1) or the glass-type terminal (see FIG. 34) isused as an example of an information processing apparatus that providesan instruction for an operation on an object on the basis of a gesture.However, the information processing apparatus may be a tablet terminal,a so-called digital camera, a toy such as a gaming device, a watch-typewearable terminal, or a contact lens with a built-in camera, forexample, that includes a camera for image capture, a display section,and a communication section. The contact lens with a built-in camera isan example of the wearable terminal, and is also called a “smart contactlens”. The smart contact lens includes a camera, an antenna, and anelectronic component (such as a transmission/reception circuit and asignal processing circuit), for example, disposed in a lens that is usedin contact with a cornea, and is operable through blinking. Operationsfor shutter release, zooming, aperture setting, and focusing, forexample, may be performed.

In the exemplary embodiments discussed earlier, the smartphone 200executes a process of receiving motion of a part of the body of the usercaptured in an image by the smartphone 200 as an operation. However, thesmartphone 200 may be used as a unit that captures an image of motion ofa part of the body, and an external apparatus (e.g. a server) mayexecute a process of receiving such motion as an operation.

In the exemplary embodiments discussed earlier, the image formingapparatus 100 (see FIG. 1), the door 400 (see FIG. 19), and the safe 500(see FIG. 31) are provided as examples of the operation target object.However, examples of the operation target object may also include acontrol column, a joystick, a shift lever, a blinker lever, a brakelever, a vehicle controller, a lighting fixture, an air-conditioner, anaudio device, and a home electric appliance.

In the exemplary embodiments discussed earlier, the image formingapparatus 100 (see FIG. 1) is an apparatus that prints a two-dimensionalimage on a recording material such as paper. However, the image formingapparatus 100 may be a three-dimensional printer that forms astereoscopic image (three-dimensional image).

The foregoing description of the exemplary embodiments of the presentdisclosure has been provided for the purposes of illustration anddescription. It is not intended to be exhaustive or to limit thedisclosure to the precise forms disclosed. Obviously, many modificationsand variations will be apparent to practitioners skilled in the art. Theembodiments were chosen and described in order to best explain theprinciples of the disclosure and its practical applications, therebyenabling others skilled in the art to understand the disclosure forvarious embodiments and with the various modifications as are suited tothe particular use contemplated. It is intended that the scope of thedisclosure be defined by the following claims and their equivalents.

What is claimed is:
 1. An information processing apparatus that is usedin contact with a user, comprising: a detection unit that detects amotion of the user in a real space performed without contacting anobject that is present in the real space; and a control unit thatinstructs the object to execute a specific function in a case where aplurality of motions by users for a common object are detected, thespecific function being correlated with the plurality of motions.
 2. Theinformation processing apparatus according to claim 1, wherein theplurality of motions are different operations prescribed in advance. 3.The information processing apparatus according to claim 2, whereinallocation of operations required from individual users differs inaccordance with a combination of the users.
 4. The informationprocessing apparatus according to claim 2, wherein the plurality ofmotions associated with execution of the specific function are set foreach specific function.
 5. The information processing apparatusaccording to claim 1, wherein the plurality of motions are correlatedwith different operation elements associated with the specific function.6. The information processing apparatus according to claim 1, whereinthe plurality of motions are set for each object.
 7. The informationprocessing apparatus according to claim 1, wherein the plurality ofmotions are correlated with the same operation element associated withthe specific function.
 8. The information processing apparatus accordingto claim 1, wherein, in a case where directions of operationscorresponding to the plurality of motions are opposite to each other,the specific function is executed in accordance with a result ofcombining the plurality of motions.
 9. The information processingapparatus according to claim 8, wherein the control unit combines theplurality of motions after adjusting magnitudes of the plurality ofdetected motions among the users.
 10. The information processingapparatus according to claim 1, wherein, in a case where directions ofoperations corresponding to the plurality of motions are the same aseach other, the specific function is executed in accordance with aresult of combining the plurality of motions.
 11. The informationprocessing apparatus according to claim 1, wherein the user is notifiedof presence of an operation of the common object by a different user.12. The information processing apparatus according to claim 11, whereinan image that indicates a motion of the different user is displayedtogether with an image for work in which the object is captured.
 13. Theinformation processing apparatus according to claim 11, wherein an imagethat is used by the different user for an operation is displayedtogether with a screen in which the object is captured.
 14. Theinformation processing apparatus according to claim 1, wherein a tactilefeel that matches the detected motion is generated and applied to theuser.
 15. The information processing apparatus according to claim 14,wherein the tactile feel differs in accordance with the specificfunction.
 16. The information processing apparatus according to claim14, wherein the tactile feel is applied in a case where an operation forthe specific function is received.
 17. The information processingapparatus according to claim 14, wherein the tactile feel is generatedin accordance with a motion of a different user.
 18. The informationprocessing apparatus according to claim 17, wherein the tactile feel isgenerated in accordance with the motion of the different user for acommon portion.
 19. A non-transitory computer readable medium storing aprogram causing a computer that is used in contact with a user tofunction as: a detection unit that detects a motion of the user in areal space performed without contacting an object that is present in thereal space; and a control unit that instructs the object to execute aspecific function in a case where a plurality of motions by users for acommon object are detected, the specific function being correlated withthe plurality of motions.
 20. An information processing apparatus thatis used in contact with a user, comprising: detection means fordetecting a motion of the user in a real space performed withoutcontacting an object that is present in the real space; and controlmeans for instructing the object to execute a specific function in acase where a plurality of motions by users for a common object aredetected, the specific function being correlated with the plurality ofmotions.